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ABSTRACT 

This booklet reports the results of a recent National 
Science Foundation study of the impact of mandated testing programs 
on curriculum and instruction in elementary and secondary mathematics 
and science education, especially the impact on teachers with large 
percentages of minority students. The study included three strands: 
(1) an item~by-item analysis of the most widely-used standardized 
tests and textbook tests in mathematics and science for grades 4, 8, 
and selected high school subjects; (2) a nationwide questionnaire 
survey responded to by 2,229 mathematics and science teachers in 
grades 4-12; and (3) interviews with 200 teachers and 100 
administrators in 6 urban districts in 6 states nationwide. Included 
is a figure titled "Percent of items testing types and levels of 
thinking 1 ' and two tables titled "Percent of teachers reporting 
preparation for mandated standardized tests (nationwide survey of 
2,229 teachers)" and "Percent of teachers with positive and negative 
views of mandated testing programs (interviews with 200 mathematics 
and science teachers in 6 urban districts)." Impact on minority 
students and other issues are discussed. The last page includes an 
order form for this and other reports. (MKR) 
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About this report This booklet reports the results of a recent National Science Founda- 

tion study of the impact of mandated testing programs on curriculum 
and instruction in elementary and secondary math and science educa- 
tion, especially the impact on teachers with large percentages of 
minority students. 

The study included three strands — 

• an item-by-item analysis of the most widely-used 
standardized tests and textbook tests in math and science 
for grades 4, 8, and selected high school subjects; 

• a nationwide questionnaire survey responded to by 2229 
math and science teachers in grades 4-12; 

• interviews with 200 teachers and 100 administrators in six 
urban districts in six states nationwide. 



Project members: George F. Madaus (principal investigator), Mary Maxwell West 
(co-principal investigator), Maryellen C. Harmon, Richard G. Lomax, Katherine A. 
Viator, Claudette Fongkong Mungal, Patricia A. Butler, Ceasar McDowell, Rebecca 
Simmons and Elaine Sweeney. 

The findings and conclusions expressed in this report do not necessarily reflect the 
position or policies of the National Science Foundation. 



Design by Joyce Hempstead. 
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Betty: Perspectives 
of an urban teacher 



Betty is a fifth grade teacher in an inner city school. Her students are 
primarily African- American and Hispanic; 85% are eligible for the free 
or reduced-cost lunch program. 



She has been teaching for 8 years; this is her second year at Madison, 
a K-6 elementary school. She taught 4 years before at a nearby middle 
school. She was trained at the local state college and has a certificate 
in elementary education. 

In math, Betty's students are subject to state-wide testing with "high 
stakes." In every school, a certain percentage of students must pass a 
cutoff score on a standardized test. If only one school fails, the entire 
district fails state requirements, which may result in takeover of the 
district by the state. The district has been borderline in passing for 
several years. 

Kay, the interviewer, asked Betty whether she uses a curriculum 
guide for math: 

I'm glued to [the curriculum guide J and to children passing the 
[state-required standardized test]. In the past few years I have 
been using the curriculum guide to identify objectives in order to 
teach to the test The teaching objectives are strictly related to 
the test Ym not teaching the items — the test is secure — but 
teaching the objectives, which are structured to the test 

...But teaching to the test leaves little time to bring in things — 
connect things. I can't branch out in the way I would like — J , 
am working on a strict time-table to cover all of the objectives. 

Kay: What would you do differently if you had more time? 

Betty: Abolish the time-table, which is related to taking creativ- 
ity out of education. The test objectives limit our ability to bring 
in outside matter. ...Lack of money is another major problem. 

...If I had more time I would teach at my own pace and at the 
pace of the students. ...If I didn't have the strict schedule for the 
test I would stick with an area until the students got it. 

Although she is not deeply familiar with the new approaches to 
science or math teaching now being recommended by curriculum 
reformers, her views mirror basic principles of these movements. 
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Kay: Can you explain more what would you do if you had more 
time and money? 

Betty: More hands-on work, in science especially. Science is 
theory — it is all well and good to teach this but students need 
to see it work, they need the model. I would also bring in outside 
speakers. 

She goes on to say that, while time is a constraint, the district's lack of 
money and the disadvantages that her students suffer are equally im- 
portant constraints to doing what she would like. Her students come 
to her poorly prepared. But she does not see the testing program as 
helping — with her teaching, or their learning. 

When asked for any other concerns or opinions about the state-required 
standardized test, she replied: 

/ would like to see standardized tests done away with. The 
problem with the testing furor is that we place so much 
emphasis on tests — in the end all the tests place judgment. 
This intimidates the students. When we use the [standardized 
test] to group students, children are aware of this. It is used as a 
measuring stick for success. 

In math, Betty follows the textbook for 95% of her teaching. She likes 
the text because "it has few errors" and it is well aligned with the cur- 
riculum objectives and the state test. She uses chapter tests and other 
tests from the textbook and extra booklets supplied by the publisher. 
About 90% of the tests and quizzes she gives are publisher-supplied, 
and 10% are made up by her. 



What's on math and Betty's state test is like standardized tests used nationwide. It is one 
science tests used of six standardized tests analyzed in this study. In the six standard- 

nationwide? ized math tests most widely used in the U.S., 97% of the questions 

tested low-level conceptual knowledge and 95% tested low-level think- 
ing. Science tests were somewhat better but still low on high-level 
knowledge and thinking (see Figure 1). 
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Figure 1: Percent of Items Testing Types and Levels of Thinking 1 
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'The most widely-used standardized tests and textSook series were selected for grades 4, 8, and the most commonly studied high school subjects. 

All items were classified on 3 dimensions: conceptual knowledge, level of thinking, and procedural knowledge. For example, of the 1 627 stan- 
dardized math items, 1 00% measured conceptual knowledge; 89% measured procedural knowledge, while 1 1 % did not. Of the 1 627 items, 95% 
measured low level thinking, and 5% measured high level thinking. 

Definitions of types and levels of thinking were derived from recent curriculum standards and research in math and science. Definitions and ex- 
amples are detailed in the Executive Summary, Appendix B or C (see list of reports available). For example Jiioli level CQDCfiBlua! knowledge in 
math or science is evidenced by ability to paraphrase the definition of a concept, generate examples and nonexamples, use models to represent 
concepts, recognize meanings and various interpretations of concepts, identify critical properties of a given concept, compare/contrast concepts, 
and apply concepts (as opposed to applying algorithms). Hiflh level thinking refers to reasoning and other "higher order thinking skills" (see reports 
for definition) and low level refers to recall and routine applications. In math and science, conceQlual knowledge and M of thjpKing were catego- 
rized as either high or low. The definition nf procedural knowledge differs in math and science, referring to use of algorithms in math (low level) or 
ability to generate, extend or justify procedures (high level), but to the complex of experimental knowledge and procedures in science. Procedural 
knowledge in science was categorized as present or absent (tested or not tested); in math, it was categorized as high, low or absent. 

Percentages shown were computed by pooling test items across grade levels and across all six publishers. See other reports for breakdown by 
grade level, content areas, and publishers. 

Number of items examined: Math: standardized tests = 1627 textbook final tests = 1423 

Science: standardized tests = 708 textbook final tests = 1046 
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The study also examined tests in the textbook series most widely used 
nationwide. The study found that textbook tests (including end-of- 
chapter tests and final course tests) are similar to standardized tests 
in content, format, and types of thinking. Again, low-level thinking 
and knowledge predominate. The comparison of types of thinking in 
standardized tests and final textbook tests in both math and science is 
shown in Figure 1. 



How are test scores Nationwide, students' scorr .n mandated standardized tests are used 
used nationwide? by administrators for student placement or promotion, teacher evalua- 

tion, and school or district evaluation. Specific uses affecting students 
are placement in remediation, special services, advanced classes or 
other ability tracks, and promotion/retention, or high school gradua- 
tion. In the nationwide survey, from 17% to 57% of math teachers 
reported that scores on mandated standardized math tests were Very 
important" or "extremely important" for student placement, student 
promotion/graduation, teacher evaluation, and school/district evalua- 
tion. Variation depended on type of use and grade level. In science, 
such reliance on test scores ranged from 13% to 50%. 

In interviews we found that, in districts and schools with less empha- 
sis on standardized test scores, and without such severe penalties 
attached, teachers felt more at liberty to adopt innovative curriculum 
materials designed to support higher-order thinking, and were doing so. 

As for textbook tests, the survey found them typically used every one 
to three weeks. These test scores often determine a large portion of 
students' term grades, and grades in turn influence decisions about 
student promotion and placement. 
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Impact on minorities In the nationwide survey, teachers with high percentages of minority 

students (in a particular class) reported more reliance on standardized 
tests for the purposes above than did teachers with few minority stu- 
dents. They also felt more pressure: of teachers who had mandated 
tests in their subject, about 75% of both math and science teachers with 
high-minority classes reported pressure from their district to improve 
standardized test scores, in comparison with about 60% of teachers 
with low-minority classes. High-minority classrooms had more low-in- 
come children (as indicated by children receiving Chapter 1 services), 
and were more often located in urban settings. 

In the nationwide survey, teachers of high-minority classrooms 
reported test-oriented teaching practices more often than teachers of 
low-minority classrooms (see Table 1). For example, in high-minority 
classrooms, about 60% of the teachers reported teaching test-taking 
skills, teaching topics known to be on the test, increasing emphasis on 
tested topics, and beginning preparation more than a month before the 
test. These practices were reported significantly less often in low-mi- 
nority classrooms. 

Like Betty, teachers with "high stakes" tests often voiced negative 
views of test validity, and of the effects of their testing programs on 
student learning and on motivation of both teachers and students (see 
Table 2). Among 200 teachers interviewed in our six urban districts, 
all math teachers and 80% of science teachers had at least one test 
that was heavily used for one or more of the purposes described on 
page 4. Sixty percent of math teachers and 63% of science teachers 
described negative effects of the testing program on curriculum or stu- 
dent learning. They stated, for example, that the mandated testing 
caused narrowing and fragmenting of the curriculum, limited the na- 
ture of thinking, or forced them to rush too much for students to learn 
well. Of 17% who mentioned positive effects on curriculum or learning, 
most cited mere "coverage" of curriculum. Two-fifths of math teachers 
and one-fifth of science teachers noted negative impact on student or 
teacher motivation. Only about 10% mentioned any positive impact of 
testing programs on motivation. 
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Table 1: Percent of teachers reporting preparation for mandated standardized tests 1 
(nationwide survey of 2229 teachers) 



High-minority class Low-minority class 
Minority 2 presence More than 60% Less than 10% 

of students of students 
are minorities are minorities 

How do you prepare your students 3 for the mandated standardized test you indicated? 



Teach test-taking skills 


math 


84 


55 




science 


75 


42 


Encourage students to work 


math 


61 


49 


hard on tests 


science 


74 


50 


Use test- motivating materials 


math 


57 


19 




science 


64 


22 


Teach topics known to be on te3t 


math 


60 


37 




science 


60 


19 


Provide test- specific material 


math 


38 


12 




science 


29 


6 


Provide similar items 


math 


61 


31 




science 


34 


13 


More than 20 hrs of preparation 


math 


31 


9 




science 


21 


4 


Pi*ep. month or more before test 


math 


74 


32 




science 


49 


39 


Use state/district test prep, materials 


math 


14 


7 




science 


11 


6 


Use practice tests 


math 


32 


12 




science 


20 


10 


> what extent 4 does the mandated standardized test influence you and your teaching to, 


Include topics not otherwise taught 


math 


58 


42 




science 


71 


33 


Exclude topics otherwise taught 


math 


39 


20 




science 


41 


16 


Increase emphasis on certain topics 


math 


71 


46 




science 


67 


38 


Decrease emphasis on certain topics 


math 


49 


25 




science 


47 


26 


Alter content of teacher-made tests 


math 


45 


32 




science 


53 


25 


Alter format of teacher- made tests 


math 


51 


38 




science 


48 


26 



'Statistically significant differences between high-minority classes and low-minority classes are shaded (p< .05). 2 "Minority" is defined accord- 
ing to the categories developed by the U.S. Office of Management and Budget (0MB Directive 15) and commonly used in government statis- 
tics. They include African-American, Hispanic, Asian/Pacific Islander, and Native-American/Alaskan. ^Teachers with more than one class 
were instructed to select their first Monday class. 4 Extent of influence indicated by these teachers was "some" or "a lot". 



Table 2: Percent of teachers with positive and negative views of mandated testing 

programs 1 (interviews with 200 math and science teachers in 6 urban districts) 



Type of view Negative Positive 



View of impact on curriculum and learning 


math 
science 


60 
63 


16 
17 


View of impact on student or teacher motivation 


math 
science 


43 
18 


11 
10 


View of test validity 


math 
science 


45 
73 


8 
3 



Percentages indicate the percentage of teachers interviewed who made one or more comments on these topics. Comments on these topics were 
compiled from various interview questions about instruction and assessment rather than from any one specific question. 



Issues The tests commonly taken by students — both standardized tests 

AND textbook tests — emphasize low-level thinking and knowledge. 
Since textbook tests and standardized tests are similar, the presence 
of low-level thinking extends beyond the instructional time spent pre- 
paring for state and district mandated standardized tests. State and 
district tests, and textbook tests, mutually reinforce an emphasis on 
low-level thinking and knowledge. These tests do not meet current 
recommendations of science and math curriculum experts nationwide. 
Their uses and effects are negatively viewed by many teachers who 
work under the mandate of "high stakes" testing programs. 

There is more reliance on mandated test scores in high-minority class- 
rooms than in low-minority classrooms. Teachers of high-minority 
classes report more test pressure and test-oriented instruction than 
teachers of low-minority classes. Yet such teachers do not believe that 
these testing programs benefit curriculum, instruction, or learning. 

These results suggest a gap in instructional emphases between high 
and low-minority classrooms that conflicts with our national concern 

° 7 in 
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for equity in the quality of education. The results also suggest a gap 
between district/state testing policies and teachers' own ideas about 
good educational practice. The latter gap conflicts with our national 
concern to honor local control in general, and teachers' professional 
judgment in particular. Both of these conflicts need attention as we 
strive to reform curriculum, assessment, teaching, and schools. 
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